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Executive Summary 


Through Delaware's Positive Behavior Supports initiative (DE-PBS), over one hundred public 
schools in Delaware have adopted the Schoolwide Positive Behavioral Supports approach to 
schoolwide discipline (Sugai & Horner, 2009], This approach focuses on developing a school-wide 
system of strategies to reduce behavior problems and foster a positive school climate. DE-PBS is a 
collaborative effort between the Delaware Department of Education (DDOE), the Center for 
Disabilities Studies at the University of Delaware, and Delaware public schools. The project is led by 
a PBS Core Team consisting of project staff members from the Center for Disabilities Studies (a 
project co-director and a research associate], the Delaware Department of Education (a DDOE staff 
member who serves as co-director), University of Delaware faculty partners (two professors in the 
School of Education who serve as consultants), and a consulting school psychologist who works in 
the public schools. In previous years, the Core Team has focused on increasing the number of 
schools implementing PBS and developing staff development materials. Although PBS entails 3 tiers 
of supports and interventions (Tier 1 for all students, Tier 2 for at-risk students, and Tier 3 for 
students with the most serious behavior problems), the primary focus of DE-PBS has been at Tier 1 
- the universal, schoolwide level. 


The number of DE-PBS schools has more than doubled in the past five years. Although many more 
schools are now designated as PBS schools, the rapid increase was associated with variability in the 
quality of program implementation across schools. To address the implementation issues, and 
increase the likelihood that the program will have the desired effects, the project has focused on 
three critical tasks in recent years. These tasks are: (1) developing key features of DE-PBS, 
including the valued student outcomes and the processes for obtaining them, (2) creating reliable 
and valid evaluation measures of those outcomes and processes, and (3) aligning key features and 
assessment measures with staff development and training. DE-PBS is currently in a state of 
transition - evolving from a primarily behaviorally-oriented approach to managing student 
behavior to a more comprehensive approach to schoolwide discipline designed not only to help 
manage student behavior but also to develop self-discipline and foster positive school climate. 

Evaluation Questions 

Two questions guided the evaluation. The first is formative, intended to help the DE-PBS Core Team 
in its current transition. The second looks again at the outcome of disciplinary infractions, using a 
different analysis than that used by Bear and colleagues (2009) to see if the results are any 
different. The specific questions are: 

(1) What are contributors and barriers to schools' implementation of DE-PBS? 

(2) What is the relationship between DE-PBS and changes in incidence of disciplinary infractions? 

Contributors and Barriers to DE-PBS Implementation 

To learn about contributors and barriers to PBS implementation, a multiple case study design with 
six schools was used. To be considered for inclusion in the study, schools had to: (1) be an 
elementary or middle school implementing DE-PBS for at least three years, (2) have the approval of 
district leaders to participate, and (3) be at either the high or low end of the continuum of school 
disciplinary infractions. We purposively selected schools at the ends of the continuum to ensure a 



diversity of experiences in the implementation of PBS. The final sample includes three middle 
schools and three elementary schools, at least one school from each county, and two schools with a 
high rate of disciplinary infractions and four with low rates. The participating schools are not 
identified because the focus is on the cross-cutting findings on barriers and contributors, rather 
than on successes or challenges at specific schools. 

At each school, administrators serving on the PBS team participated in individual interviews. All 
members of a school's PBS team were invited to participate in a focus group interview. Focus 
groups were also held with teachers and, in the schools with low rates of disciplinary infractions, 
with students. (Given time constraints, focus groups could not be held with students at all schools.] 
In all but one school, participants in the teacher and student focus groups were randomly selected 
from lists provided by the schools. (One principal chose to select teachers and students by following 
our random selection procedures.] With the exception of the students, all interviewees were asked 
to complete an online version of the Delaware Assessment of Strengths and Needs for Positive 
Behavior Supports (Bear, Burwell, Baker, Blank, & Boyer, 2009], which includes items aligned with 
features of DE-PBS and drawn from the literature. Respondents from schools with higher incidence 
of disciplinary infractions were more likely to give their school low ratings than respondents from 
schools with lower incidence of infractions. These results indicate the diversity of the selected 
schools on dimensions relevant to PBS implementation, and thus confirm the schools’ likely utility 
in providing different perspectives on DE-PBS. 

The data from the individual and focus group interviews were transcribed from audio recordings 
and then analyzed using a combination of deductive categories drawn from the key features of DE- 
PBS and inductive coding to identify emergent themes. In keeping with the focus on cross-cutting 
contributors and barriers rather than on the experiences of individual schools, the analysis was 
conducted by type of respondent first (that is, all PBS team focus groups, then all administrators, 
and so on] and then the results were aggregated across the different respondents to identify major 
categories and specific contributors and barriers. This approach limits the opportunity to include 
the role of school context in the analysis. The goal of this part of the evaluation was to help the team 
learn how to improve school’s implementation of PBS, whatever its context. 

Findings 

Four dominant themes related to contributors and barriers to the implementation of DE-PBS were 
identified: 

(1) consistency and adaptability, 

(2) rewards, 

(3) data-based decision making, and 

(4) professional development and support. 

These themes encompass both contributors and barriers to PBS implementation. For example, the 
theme of consistency and adaptability includes the positive aspect of PBS evolving over time in 
response to changing needs, and the concern about PBS becoming stagnant and unresponsive. 



Consistency and Adaptability 


Consistency and adaptability, the most prominent factor identified in the data, has three 
dimensions: the consistency of commitment to PBS over time; the consistency with which school- 
specific PBS policies and practices were applied within each school; and the adaptability of PBS as 
part of a dynamic and evolving school culture. Table 1 provides illustrative quotes for the 
contributors and barriers associated with this theme. In this and the other tables in this section, the 
initials in brackets after each quote refer to the source of the quote, with C = coach, A = 
administrator, T = teacher (not a member of the PBS team], S = student, and TM = PBS team 
member. 


Table 1. Contributors and barriers in the consistency and adaptability 


Contributors 

Illustrative Quotes 

Consistent commitment to PBS by 
teachers and administrators 

"They have to have buy-in - that’s important so you have to get people 
to understand why it’s important and why you’re doing it." [A] 

Consistency of policies and 
practices as applied in the school 

"Those rules are throughout the whole school, not only in the 
classroom, but it’s being safe in the hallway, in the playground, on the 
bus, and in the cafeteria. ..The rules are the same no matter where you 
are and no matter what grade you are in." [T] 

Willingness to adapt and 
experiment within the framework 
of PBS 

"So I think one thing we’re always doing is trying to say like 'where are 
we?’ and 'what can we do to improve it?”’ [A] 

Barriers 

Illustrative Quotes 

Lack of buy-in to PBS 

"There are still considerable people, teachers and staff, who are 
resistant and that clearly affects how well they are able to implement. 
There are some naysayers that clearly don’t want to be on board." [C] 

Limited understanding of PBS 
principles 

"...something we’re constantly working on is people understanding it [the 
currency used as rewards] has to be one at a time..That's an education 
that goes on every month it seems like. Being sure that people use them 
the way they're supposed to be used." [A] 


Rewards 

When asked to describe the major components of PBS in their school, almost all respondents 
mentioned the rewards system before any other element of the program. The structure of rewards 
systems was similar across the schools. Each school had a currency system in place wherein 
students were issued varying amounts of the school currency for different types of positive 
behavior. Although the currency was a common element, there was variation in how it was used. In 
different schools, students could use the currency to buy pre-priced items in a PBS store, bid on 
items at auctions, participate in a weekly lottery, or purchase extra privileges. In some cases, 
schools used a combination of group activities and events with individual prizes. In the six schools 
that we studied, rewards seem to dominate the definition of PBS. 


The contributors and barriers associated with the theme of rewards are listed below, with 
illustrative quotes for each. 




Table 2. Contributors and barriers in the rewards theme 


Contributors 

Illustrative Quotes 

Student input 

"With the school store it has changed because of the survey that was 
taken by fifth grade students and they came up with different things they 
would like to see in the school store." [T] 

Appropriate nature and timing of 
rewards 

"Another thing that is always told to us is to keep the rewards simple 
because you don't want the students thinking that if I'm good 1 can get a 
reward. You want them to think 1 should do the right thing. So we always 
try to keep the rewards really simple." [TM] 

External support 

"Another example of a donation or a pitch would be to go to the local 
WaWa and students that have the best behaved classroom. ..for that 
month, they would receive a free milkshake from WaWa." [A] 

Barriers 

Illustrative Quotes 

Lack of clarity/ agreement about 
when to use rewards 

"...1 feel as though not every teacher gives the students the tickets for the 
same behaviors. It's not consistent among teachers." [TM] 

Student manipulation of the system 

"1 just feel that as the kids are going from grade to grade they’re 
learning how to manipulate the system, so by fifth grade I mean I’ve 
had a kid come up to me and say put his feet on my desk and say let’s 
make a deal.” [T] 

Rewards that are inappropriate for 
the age group 

"The rewards are more for like middle school interests. Like in middle 
school they’re not going to want to be winning candy, they’re going to 
want to win money and stuff that they can actually use." [S] 


Data-Based Decision Making 

Data-based decision making is a key feature of PBS and was addressed specifically in our interview 
questions. In response, teachers and administrators described positive aspects of using data, but 
also identified barriers to the use of data. Both positive aspects and barriers, with illustrative 
quotes for each, are listed in the following table. 


Table 3. Contributors and barriers in the data-based decision-making theme 


Contributors 

Illustrative Quotes 

Responding to emerging issues 

"Say we noticed in February that the playground behavior was poor, it 
was going down and we were having incidents, then we would go back 
and discuss with our co-workers what we need to do, possibly go over 
the rules, do some more skits, remind the kids again about the PBS 
rules, show them again what they look like, and model again good 
behavior and what we’re expecting out of them." [TM] 

Monitoring individual behavior 

"We look at our monthly offenses (level 2, 3, 4). We look at where they are 
occurring, are there repeat offenders, and what have we done for those 
repeat offenders. Are they seeing the guidance counselor, so they have an 
FBA (Functional Behavior Assessment), do they have an IEP that has 
accommodations. Are we meeting those?" [A] 

Increased staff buy-in 

"But some of our teachers weren't buying into it as much as they could 






have for a lot of reasons. One is because they didn't see the benefits of it. 
Well, now they see the benefits of it because we are able to track it and 
get a little bit better, and we are able to use it a little bit better, so now we 
have more teachers buying into it." [A] 

Barriers 

Illustrative Quotes 

Data as window-dressing 

"...we have the academic data posted all around in our classroom, down 
stairs, when you walk in the buildings, you hear about it during staff 
development meetings, but we never hear about the school climate data. 
And the referrals, that data is not posted." [T] 

Unsystematic collection of data 

"1 would ask them - where's your data? First, it was not even there to 
begin with. Then they started bringing data, but 1 think that they thought 
all they had to do was look at it instead of analyze it. Now 1 think they are 
getting it; it's emerging. 1 think they're improving, but that has always 
been a major roadblock in their ability to reflect on what they're doing and 
make decisions based on the data about what needs to be done 
differently." [C] 


Professional Development and Support 

Schoolwide and targeted subgroup professional development and support was a theme discussed 
among all adult participant groups. Need for professional development varied widely across 
schools and refresher training was identified as an important part of the process. Support from the 
district coach also varied with some schools reporting that they needed little assistance and others 
needing more. The quotes below illustrate the contributors and barriers related to professional 
development and support. 


Table 4. Contributors and barriers in the professional development and support theme 


Contributors 

Illustrative Quotes 

Awareness of District Coach as a 
resource 

"At this point we felt we haven’t needed our district coach, she’s there if 
we need her, but things have been running smoothly. " [TM] 

Annual refresher training 

"We actually revamped our program. ..and so we went to a retreat where 
we spent a good part of the day re-training all the new staff so everybody 
had a refresher." [A] 

Barriers 

Illustrative Quotes 

Insufficient training for new 
teachers 

"I'm a first year teacher here and I've never gotten any training. 1 spent the 
first three weeks, they gave me tickets [reward currency], and 1 had no 
idea what they were." [T] 

Lack of refresher training 

"1 think it is necessary every year to cover the whole thing - the basics, the 
philosophy, the way it works. ..There is some time that should be invested 
in making it work every year and we just haven't been able to put the time 
into because of all the other constraints in the building." [T] 

Limited resources 

"That is how the District could help us - ...provide substitute teachers so 
that you know that everyone can get the state wide training that PBS 
offers." [TM] 





Recommendations 


Specific recommendations can be drawn from the lists of contributors and barriers described 
above. In addition, some cross-cutting recommendations can be identified: 

(1) Teacher commitment to the initiative needs to be developed and reinforced. Communication of 
guidelines and the rationale for practices could facilitate teacher buy-in. Systematic use of data 
to demonstrate the effectiveness of PBS practices was also linked to teacher buy-in. 

(2) Clear implementation guidelines should be provided to all school staff through a structured 
system of professional development and information dissemination. The guidelines need to help 
teachers respond positively to an often diverse population of students. They also need to 
encourage consistent and appropriate implementation of the rewards system based on a shared 
understanding of PBS in the school. 

(3) Systematic data collection and use allows teachers and administrators to monitor the nature, 
location, and frequency of smaller disciplinary infractions before they become larger issues. 

(4) Including student input in data-based decision-making can be useful in addressing barriers 
associated with the rewards system, including selecting appropriate rewards and monitoring the 
consistency with which rewards are offered. 

(5) PBS orientation/training for new and substitute teachers could increase the consistency of 
application within the school. Similarly, refresher training may be needed for more experienced 
teachers. However, given the limited resources for professional development and the 
experience of some teachers with PBS, creativity in how to provide the training - to whom, 
when, and in what format - will be needed. 


Analysis of the Relationship between Disciplinary Infractions and Schoolwide DE-PBS 

Methodology 

To answer evaluation question 2, differences in suspension rates over time in PBS and non-PBS 
schools were tested using data from 2005-2008, with separate models for elementary and middle 
schools. (Due to changes in how suspension data were reported to and collected by the state, data 
before 2005 or after 2008 were not used.] The sample was limited to elementary or middle schools 
that had participated in the student climate survey. In addition, PBS schools had to have 
participated in the PBS program for at least three years as of fall 2009. A total of 55 elementary 
schools fit the eligibility criteria, with 17 [31%] not implementing and 38 [69%] implementing PBS. 
Of the 18 middle schools that fit the eligibility criteria, four [22%] were not implementing and 14 
[78%] were implementing PBS. Using suspension rate data from these schools, a longitudinal 
hierarchical linear modeling analysis [HLM] was used to account for the nested structure of the 
data (suspension rate data nested within schools over time]. The original design planned to include 
other possible influences on discipline over time, such as prior academic performance, percentage 
of students with free and reduced lunch, school climate (as measured by The Delaware School 
Climate Survey - Student (DSCS-S], Bear, Smith, Blank, & Chen, in press], and level of PBS technical 
assistance (as measured by a survey of coaches]. Although the sample included only those schools 
that had school climate survey data, not all schools agreed to share the data for research purposes. 
As a result, climate was not included as a variable in the analysis. Incomplete data on levels of 
technical assistance meant that this variable was also not included. All other school-level discipline, 
achievement, and demographic data were provided by the Delaware Department of Education. 



Findings 

In the analyses, neither PBS nor non-PBS schools showed significant change in the number of 
suspensions over time. That is, suspension rates were not significantly increasing or decreasing. 
Neither of these results are altered by including socioeconomic status or DSTP scores as covariates. 
In addition, no differences were found between PBS and non-PBS schools in 2005 suspension rates 
in either the elementary or middle school samples. 

The lack of evidence for a relationship between PBS and suspension rates can be attributed to six 
limitations: The first limitation was the inability to use an experimental design and randomly assign 
schools to PBS implementation or no implementation. Second, the use of suspensions as an 
outcome measure is problematic because of fluctuations that occur over time that are unrelated to 
student behavior and that they do not capture positive changes in student behavior. Third, the 
criteria for being included in the sample (such as implementing PBS for three or more years and 
having approval to use student climate survey data] reduced the size of the sample and thus the 
power to detect significant differences. Fourth, the covariates that we were unable to include in the 
analysis may have helped explain why PBS and non-PBS schools do not show significant differences 
in suspension rates. Fifth, schools in the PBS group had staggered implementation start points. Due 
to variation in how suspension data were collected and reported before 2005 and after 2008, we 
are unable to account for the difference in starting dates of implementation. Sixth, the qualitative 
data indicate that the category of "PBS school" (versus non-PBS school] is not clearly defined. The 
six case study schools are all PBS schools, but PBS was implemented in varying degrees and in 
varying ways in each school. At the same time, schools that are not implementing PBS may have 
programs that are similar in nature and intent to PBS. Thus, a quantitative comparison of PBS and 
non-PBS schools may have been conceptually flawed from the beginning. 

Recommendations 

Some of the issues that made these analyses problematic could be resolved with the following 
changes: 

(1) Sample size was a major limitation of this analysis. Studies involving more schools may provide 
more reliable estimates of direct effects. 

(2) The change in the kind of infraction data maintained by DDOE limited the ability to do 
longitudinal analyses over more than 3 years. Using more stable outcome measures would make 
it easier to examine how being a PBS school is related to intended outcomes over time. (For 
example, over time, does a PBS school have an improved school climate, or reach a threshold 
level of positive school climate?) 

(3) Analyses using the outcomes that are likely to be most directly affected by PBS practices might 
be more sensitive. Past research (Bear, 2010) has suggested that suspension data are highly 
related to office referrals. As noted previously, there are limitations to the use of suspension 
data. For example, the case studies here suggest that there is some manipulation of the 
suspension and office referral numbers by in-school practices discouraging referrals. However, 
there may be other outcomes that are appropriate indicators of PBS effects. 

While these recommendations respond to some of the major flaws of our analyses, they do not 
address the more fundamental issue of the definition of "PBS" and what it means to be a PBS school 
in Delaware. 

Implications for Future Evaluations 

The findings from both the qualitative and quantitative components of the study point to next steps 
for evaluating DE-PBS. We know that among DE-PBS schools there are varying levels of 



implementation. Conversely, it is reasonable to assume that some schools that have not officially 
adopted PBS are implementing practices consistent with PBS approaches. If this is true, a more 
appropriate comparison would be between schools that implement practices consistent with PBS 
and those that do not. This suggests two follow-up studies. The first would develop a rubric (or test 
of existing measures] to rate schools on how well it aligns with PBS principles. The second study 
would then build on the first with a quantitative analysis of effects on suspension rates (or other 
appropriate outcome measure] on randomly selected schools if possible, using the degree of fidelity 
with PBS principles — rather than whether the school was labeled as PBS or non-PBS — as the 
independent variable. In addition, the model could be used to explore whether additional school 
characteristics explain any differences in outcomes. 

Implications for DE-PBS 

The report closes with a postscript reflection from a PBS expert and member of the DE-PBS 
leadership team. The postscript discusses the evaluation results in light of specific program 
elements and programmatic efforts in Delaware, prior evaluation work conducted in Delaware, and 
includes a set of six recommendations for the future. The recommendations focus on specific steps 
the DE-PBS leadership team can take to facilitate improved program implementation through 
additional professional development, use of new instrumentation grounded in the Delaware PBS 
key features, and the strategic selection of schools for more intensive work. 



